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1.  SUMMARY 

Vision.  Just  as  flight  simulators  enable  pilots  to 
safely  practice  responses  to  emergencies,  the 
challenge  now  is  to  develop  virtual  environment 
technology  for  the  training  together  of  small  teams  on 
foot — military  squads,  Coast  Guard  boarding  parties, 
police,  EMTs,  emergency  room  trauma  teams,  hazmat 
teams,  etc.  Such  training  allows  repeated,  varied 
practice.  The  goal  is  you  are  there;  you  learn  by 
doing  with  feedback;  you  jell  as  a  team  by  doing 
together.  First,  we  must  clearly  envision  what  is  wanted.  This  we  will  call  the  Immersive  Team  Trainer 
(ITT). 

The  successes  of  flight  and  ship  bridge  simulators  encourage  us.  Their  use  for  training  mounted  teams  is  a 
well  understood  and  trusted  accomplishment.  Decades  of  development  have  brought  flight  simulators  to 
mature  excellence;  simulators  for  training  other  vehicle  crews  are  rapidly  approaching  this  maturity.  Such 
simulators  have  been  proven  to  be  not  only  more  cost-effective  per  hour  than  live  vehicle  training,  but  more 
effective  as  well,  since  VEs  can  provide  a  higher  density  of  experiences  and  the  chance  to  practice  rare  and 
dangerous  scenarios  safely.  The  vision  is  to  extend  VEs  for  training  dismounted  teams  effectively  and  cost- 
effectively.  This  has  not  been  done  yet  because  it  is  technically  much  more  difficult  than  immersive  training 
for  vehicle  crews. 


Brooks,  F.;  Fuchs,  H.;  McMillan,  L.;  Whitton,  M.;  Cannon-Bowers,  J.  (2006)  Virtual  Environment  Training  for  Dismounted  Teams  -  Technical 
Challenges.  In  Virtual  Media  for  Military  Applications  (pp.  22-1  -  22-10).  Meeting  Proceedings  RTO-MP-HFM-136,  Paper  22.  Neuilly-sur-Seine, 
France:  RTO.  Available  from:  http://www.rto.nato.int/abstracts.asp. 


RTO-MP-HFM-136 


22-1 


Report  Documentation  Page 

Form  Approved 

OMB  No.  0704-0188 

Public  reporting  burden  for  the  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources,  gathering  and 
maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  this  collection  of  information, 
including  suggestions  for  reducing  this  burden,  to  Washington  Headquarters  Services,  Directorate  for  Information  Operations  and  Reports,  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington 

VA  22202-4302.  Respondents  should  be  aware  that  notwithstanding  any  other  provision  of  law,  no  person  shall  be  subject  to  a  penalty  for  failing  to  comply  with  a  collection  of  information  if  it 
does  not  display  a  currently  valid  OMB  control  number. 

1.  REPORT  DATE 

01  JUN  2006 

2.  REPORT  TYPE 

N/A 

3.  DATES  COVERED 

4.  TITLE  AND  SUBTITLE 

5a.  CONTRACT  NUMBER 

Virtual  Environment  Training  for  Dismounted  Teams  Technical 

5b.  GRANT  NUMBER 

^iianciigcs 

5c.  PROGRAM  ELEMENT  NUMBER 

6.  AUTHOR(S) 

5d.  PROJECT  NUMBER 

5e.  TASK  NUMBER 

5f.  WORK  UNIT  NUMBER 

7.  PERFORMING  ORGANIZATION  NAME(S)  AND  ADDRESS(ES) 

U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences 
Simulator  Systems  Research  Unit  12350  Research  Parkway  Orlando,  FL 
32826-3276  USA 

8.  PERFORMING  ORGANIZATION 

REPORT  NUMBER 

9.  SPONSORING/MONITORING  AGENCY  NAME(S)  AND  ADDRESS(ES) 

10.  SPONSOR/MONITOR'S  ACRONYM(S) 

1 1 .  SPONSOR/MONITOR' S  REPORT 
NUMBER(S) 

12.  DISTRIBUTION/AVAILABILITY  STATEMENT 

Approved  for  public  release,  distribution  unlimited 

13.  SUPPLEMENTARY  NOTES 

See  also  ADM002024.,  The  original  document  contains  color  images. 

14.  ABSTRACT 

15.  SUBJECT  TERMS 

16.  SECURITY  CLASSIFICATION  OF: 

17.  LIMITATION  OF 
ABSTRACT 

uu 

18.  NUMBER 
OF  PAGES 

29 

19a.  NAME  OF 
RESPONSIBLE  PERSON 

a.  REPORT 

unclassified 

b.  ABSTRACT 

unclassified 

c.  THIS  PAGE 

unclassified 

Standard  Form  298  (Rev.  8-98) 

Prescribed  by  ANSI  Std  Z39-18 


Virtual  Environment  Training  for 
Dismounted  Teams  -  Technical  Challenges 


ORGANIZATION 


Technical  Challenges.  The 

Display 

Tracking 

Locomotion 

Tools  and  Instruments 

Interaction  with  each  other 

Interaction  with  world 

Inanimate  objects 
People 

Physics 

Sound 


technical  challenges  of  such  an  extension  abound: 

Mounted  Dismounted 

Unified  for  all  observers  Distinct  for  each 

At  most,  head  position;  rarely  that  Heads  and  limbs  of  each  team  member 


Not  required 
Physical  mockups 
Speech 

Only  vehicle  controls 
Real  touch 
Speech 

Well  understood  vehicle  physics 
Unified  for  all 


Extensive,  separate  per  trainee  [Usoh,  '99] 
Both  physical  and  virtual 
Speech,  touch,  facial  expression 
Various  manipulanda  [Insko,  '01] 
Manipulation  of  physical,  virtual  objects 
Speech,  body  language 
Much  more  complex,  moving  objects 
Distinct  localization  for  each 


2.  THE  VISION 


Figure  1.  The  image  on  page  1  shows  the  emergency  situation 
as  it  really  happened.  The  inset  image  shows  the  unlit  training 
space  with  an  approximate  physical  model  of  the  ambulance. 
The  larger  image  shows  the  full  virtual  environment  used  by 
the  trainees  and  instructors  (Dwg: State). 


22-2 


RTO-MP-HFM-136 


Virtual  Environment  Training  for 
Dismounted  Teams  -  Technical  Challenges 


2.1  The  Virtual  Environment 

The  virtual  environment  itself  is  easy  to  envision  (Figure  1).  Six  image -bearing  surfaces  enclose  a  small  team, 
such  as  an  EMT  team  or  an  infantry  squad.  Each  trainee  sees,  in  stereo,  images  customized  for  his 
dynamically  changing  viewpoint.  Thus  the  enclosing  world  looks  real — providing  realistic  occlusion,  kinetic 
depth  effect,  illumination,  stereopsis. 

Each  trainee  can  see,  hear,  and  touch  his  buddies.  Every  object  he  must  handle  to  perform  his  task  is  real  or 
physically  mocked  up  [Insko,  2001];  objects  the  team  need  only  behold  are  virtual,  but  as  they  move,  their 
physics  are  realistic. 

Sounds  from  the  environment  are  properly  localized  for  each  trainee.  When  the  team  needs  to  move  beyond 
the  physical  space  available,  the  squad  leader's  walking-in-place  translates  the  real  space  in  the  virtual  world. 

2.2  The  Training — Critiqued  Practice  with  Branched  Scenarios 

The  very  tracking  necessary  for  individualizing  the  visual  displays  yields  a  superabundance  of  individual  and 
team  performance  data.  Real-time  analysis  identifies  behavior  patterns  in  trainee  speech  and  head  and  limb 
trajectories,  compares  them  to  expert  and  erroneous  behavior  patterns,  and  produces  diagnostics. 

The  trainer  can  use  these  diagnostics  for  after-action  review  and/or  can  branch  the  scenario  to  provide  more 
practice  for  flubbed  responses.  Ideally,  the  behavior  patterns  can  be  used  to  generate  automatic  after-action 
reviews  and  "pinball  scores,"  so  that  teams  can  practice  and  compete  without  trainers. 

An  automatic  scenario  generator  takes  specified  knowledge,  skills,  and  attitudes  and  generates  a  rich  variety 
of  event-branched  scenarios,  so  that  in  a  compressed  time  the  trainees  experience  many  varied  situations  that 
never  repeat. 

2.3  The  VE  System 

The  system  itself  is  transportable,  easy  to  calibrate,  built  with  commercial  off-the-shelf  computers,  projectors, 
and  trackers,  and  demands  only  the  space  of  the  training  volume  itself. 

Latency.  The  ITT  achieves  50  ms  end-to-end  latency  for  complex  virtual  environments,  whereas  most 
CAVEs  have  100-250ms  latencies.  Low  latency  prevents  most  simulator  sickness. 

Model  Acquisition.  An  advance  in  the  ITT  is  the  economical  generation  of  the  3-D  models  of  virtual  objects. 
The  ITT  developers  acquire  such  models  from  images  taken  with  video  cameras  and  laser-rangefinders. 


3.  THE  VE  TECHNICAL  CHALLENGES  IN  REALIZING  THE  VISION 

3.1  Display 

In  flight  simulators,  the  entire  cabin  crew  shares  one  display.  In  the  ITT,  each  team  member  needs  a 
personalized  stereo  view.  Yet  a  critical  part  of  the  training  concept  is  that  all  team  members  must  be 
physically  collocated,  share  physical  objects  as  well  as  visual  environment,  talk  to  each  other,  see  each  other's 
faces,  touch  each  other.  This  combination  of  collocation  and  individualized  displays  is  a  major  challenge. 

The  proposed  system  must  go  far  beyond  today’s  CAVE*-like  immersive  environments: 
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•  Front-surface  projection  radically  shrinks  the  required  floorspace.  One  must,  and  can,  solve  the 
keystoning,  depth-of-field,  and  shadowing  problems  that  have  inhibited  front-surface  projection  [Jaynes. 
'01;  Low,  '01;  Sukthankar,'01]. 

•  It  must  have  much  higher  resolution  imagery. 

•  It  must  deliver  customized  stereo  views  for  at  least  four  persons,  whereas  only  one  can  get  a  proper 
perspective  view  in  today’s  CAVEs. 

•  It  must  track  not  only  heads,  but  hands,  feet,  and  tools,  so  as  to  simulate  interactions  between  trainees 
and  virtual  objects. 

Individualized  Display  for  Each  Trainee.  Ceiling-mounted  projectors  will  paint  imagery  not  only  on  the 
walls  but  also  on  all  the  other  surfaces,  even  moving  real  objects  [Raskar,  '99;  Lok,  '03].  The  system  will 
generate  individualized  head-tracked  stereo  imagery  for  each  team  member,  and  time-shuttered  glasses  will 
filter  out  the  other  trainees’  views. 

The  farther-out  technology  vision  augments  the  projectors  with  lenticular  autostereoscopic  displays  that  show 
the  proper  views  from  every  viewpoint  simultaneously.  These  will  appear  first  on  real  objects  in  the  ITT, 
such  as  the  ambulance  in  Figure  1,  later  on  the  walls. 

3.2  Tracking 

One  must  track  the  position,  but  not  the  orientation,  of  the  eyes  of  each  trainee  in  order  to  render  his  imagery 
from  the  proper  viewpoint.  Inertial  or  outward-looking  optical  trackers  yield  the  high  required  angular 
precision  [Welch,  '01].  It  is  also  necessary  to  track  the  hands,  feet,  and  tools  of  each  trainee,  so  that  one  can 
compute  interactions  with  virtual  objects.  There  is  promise  of  new  hybrid  tracking  systems,  using  spread- 
spectrum  sound  and  imperceptible  structured  light  [Fuchs,  '05],  as  well  as  skeletal  models  fitted  to  visual 
hulls  from  video  silhouettes  [Matusik,  '00],  and  models  generated  from  reduced  marker  sets  [Liu  06]. 

3.3  After-Action  Review 

The  very  tracking  necessary  for  calculating  the  proper  behavior  of  objects  also  yields  massive  amounts  of  data 
about  the  actions  of  the  trainees.  The  system  should  capture  this  data  for  after-action  review.  This  data 
should  be  reduced  to  behavioral  descriptors  that  can  cue  instructors  for  after-action  review.  The  current  path- 
characterization  work  on  the  ONR  Virte  Project  is  a  first  step  [Whitton,  '05]. 

In  the  longer  run,  these  behavioral  descriptors  should  be  automatically  harnessed  to  yield  feedback  to  help 
team  members  train  themselves  and  each  other.  The  cost  and  logistical  complexity  of  having  instructors  is 
one  of  today’s  inhibitors  to  frequent  team-training  practice.  Even  a  single-variable  "pinball  score"  (and  a  list 
of  high  scores  to  date)  provides  feedback  and  motivation  that  enhances  training. 

3.4  Model  Acquisition 

A  substantial  cost  in  building  immersive  scenarios  is  the  development  of  models  of  the  environment  and 
objects  in  it.  Today  this  is  done  essentially  by  computer-assisted-design  (CAD)  modeling:  tedious,  time- 
consuming,  and  costly.  For  environments  that  exist,  the  models  can  today  be  economically  acquired  from 
laser-rangefinder  images  combined  with  color  photography,  and  from  moving  video  cameras  [Pollefeys,  '02]. 
These  techniques  can  also  be  used  to  acquire  models  of  separate  objects,  which  are  then  manually  combined 
into  virtual  environments. 
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3.5  Scenario  Acquisition 

Besides  the  instructor-guided  automatic  scenario  generation  described  in  Section  4.2,  we  believe  it  would  be 
very  useful  to  capture  acted-out  scenarios  by  dynamic  computer-vision  techniques.  For  scenario  generation, 
these  need  to  be  segmented  and  techniques  developed  for  piecing  segments  together  in  new  ways  to  create 
new  scenarios. 

3.6  Sound 

Hodges  and  others  have  found  that  realistic  sound  may  be  more  important  than  visual  fidelity  in  creating  the 
illusion  of  presence  [Rothbaum,  '97].  For  effective  training,  we  believe  sound  needs  to  be  generated  in  proper 
synchrony  with  visual  images,  and  displayed  properly  localized  for  trainees.  Our  experiments  indicate  that 
localization  in  azimuth  only,  as  opposed  to  3-D  localization,  may  quite  suffice  for  trainees  who  are  free  to 
move  their  heads.  We  observed  experimental  subjects  consistently  cocking  their  heads  to  determine  sound- 
source  elevation,  rather  than  relying  on  their  head  transfer  functions. 

3.7  Physics  for  Virtual  Objects 

Virtual  objects  in  the  environment  should  display  realistic  physics  in  their  interactions  with  each  other 
[Ehrmann,  '00;  Hirota,  '99;  Hoff,  '01].  When  real  objects  interact  with  virtual  ones,  the  physics  should  be  as 
plausible  as  possible,  even  though  virtual  objects  inherently  cannot  impose  forces  on  real  ones. 

3.8  Computer-Generated  People 

Often  it  is  the  people  in  the  environment  who  create  much  of  the  stress.  For  economical  team  training,  these 
need  to  be  virtual,  computer-generated  people,  with  realistic  behaviors.  The  work  of  the  Institute  for  Creative 
Technologies  is  a  great  first  step  [Rickel,  '02], 

3.9  Speech  Recognition  and  Generation 

The  trainees'  utterances  are  an  important  behavior.  Recognizing  them  and  doing  communication  analysis  on 
them  is  at  least  as  important  as  knowing  physical  behavior.  This  requires  recognizing  not  only  words  and 
larger  syntactic  patterns,  but  also  distinguishing  the  voices  of  the  several  trainees. 

The  trainees  also  will  interact  with  computer-generated  people,  so  real-time  speaker-independent  speech 
analysis  and  generation  is  a  crucial  component. 

3.10  System  Integration 

Each  individual  technological  challenge  is  great.  Making  any  of  them  work  will  be  an  important  technical 
contribution.  Integrating  them  all  into  a  single  working  system  is  an  equal  challenge  in  system  engineering 
that  must  not  be  overlooked  or  minimized  in  planning  funding  and  schedules  [Brooks,  '05]. 


4.  TEAM-TRAINING  SCIENCE  CHALLENGES 

Team-training  science  is  fundamental  to  this  concept:  it  provides  training  strategies;  it  provides  guidelines  and 
tools  for  developing  content.  Mounting  evidence  suggests  that  a  set  of  team  competencies  can  be  identified 
and  that  these  competencies  are  generalizable  across  team  task  situations  [Cannon-Bowers,  '98a, c]. 
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4.1  Why  Scenario-Based  Training  for  Teams? 

Team  training  develops  shared  mental  models  via  shared  experiences;  all  team  members  participate 
together — seeing,  talking,  touching,  and  interacting  with  each  other.  Scenario-based  training  (SBT)  relies  on 
controlled  exercises  or  vignettes,  where  the  trainee  is  presented  with  cues  found  in  the  actual  task 
environment,  performs,  is  evaluated,  and  is  then  given  feedback.  SBT  differs  from  more  traditional  training  in 
that  there  is  no  separate  formal  curriculum;  instead,  the  scenarios  themselves  are  the  curriculum  [Cannon- 
Bowers,  '98a, b].  Hence  scenarios  must  be  crafted,  and  training  executed,  so  that  it  accomplishes  specified 
training  objectives.  Effective  SBT  requires  expert  scenario  authors  and  instructors,  and  they  are  in  short 
supply.  Homeland  security  has  substantially  increased  the  demand  for  team  training,  the  skills  to  be  trained, 
and  the  uniformity  and  quality  required. 

Scenario-Based  Training  under  Stress.  Cannon-Bowers  and  others  have  demonstrated  the  power  of  SBT  in 
several  complex  operational  environments  and  completed  a  large-scale,  multi-year  study  of  decision  making 
under  stress  by  high  performance  combat  teams  [Cannon-Bowers,  '98b;  Salas,  '00].  They  found  that  exposure 
to  many  varied  task  instances  helps  develop  decision  makers  who  respond  quickly  and  maintain  situational 
awareness  while  dealing  with  ambiguity.  Immersive  simulation  can  do  this  in  a  controlled,  cost-effective 
manner. 

4.2  A  Big  Challenge  -Generating  Scenarios 

Scenarios  make  or  break  scenario-based  training.  Today  they  are  costly  and  time-consuming.  Therefore  an 
important  task  is  to  create  a  scenario  generation  software  tool  that: 

•  Generates  valid  stories  and  scenarios 

•  Is  easily  used  by  scenario  authors,  via  a  point-and-click  interface 

•  Is  flexible — accommodates  many  variables  and  input  conditions 

•  Allows  manipulation  of  scenario  difficulty  and  stressors 

•  Is  scalable  for  individuals,  teams,  and  teams-plus-autonomous  agents 

•  Is  usable  for  many  training  levels — live  drills,  VE,  tabletop  exercises,  classroom  instruction,  computer- 
based  training,  handheld  computer  games,  and  mobile  telephones 

•  Connects  to  the  performance-data  capture  tool,  the  feedback  tool,  and  instructor  aids 

•  Is  easily  updated  and  exports  scripts  to  word  processors  for  further  customization 

Build  on  an  Existing  Tool.  The  developers  should  leverage  Bowers’  team’s  experience  and  their  public- 
domain  work-product  from  aviation  training.  Through  a  Navy-FAA  partnership,  the  Rapidly  Reconfigurable 
Line-Oriented  Evaluation  (RRLOE)  scenario-generation  tool  was  developed  for  testing  aviation  proficiency  in 
commercial  pilots  [Bowers,  '97].  RRLOE  has  been  delivered  to  over  50  aviation  concerns,  including  every 
major  U.S.  airline.  Because  the  RRLOE  tool  is  generic  in  structure  and  platform,  the  software  core  can 
assemble  event  sets  for  scenario-based  training  in  other  domains.  RRLOE  strengths  include  its  relational 
database  that  directly  ties-in  qualification  standards,  task  analyses,  and  cross-environment  standardization  of 
terms. 

But  some  challenging  new  requirements  arise  for  scenarios  in  the  proposed  environments: 

•  Branching.  RRLOE  can  create  scenarios  that  target  specific  events  and  skills,  but  it  currently  does  not 
readily  handle  branching  on  data  sensed  from  the  ongoing  simulation. 
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•  Real-time  scenario  generation.  The  existing  software  does  not  yet  run  in  real  time,  which  will  be 
necessary  to  avoid  combinatorial  explosion,  once  rich  branching  is  incorporated. 

•  Interaction  with  autonomous  agents.  The  RRLOE  engine  should  be  expanded  to  incorporate 
descriptions  and  behavior  models  of  autonomous  agents. 

Other  Tools  Needed.  Training  Management  System,  Instructor  Interface  Tools,  Automatic  Performance 
Assessment  tools,  Post-exercise  Analysis  and  Feedback  tools;  Asset  Versioning  and  Management  System. 

4.3  Research  in  SBT  for  Teams. 

To  maximize  the  usefulness  of  the  ITT,  its  development  should  include  research  in  basic  learning  science  to 
generate  new  knowledge  of  team  competencies,  team  performance,  and  team  training.  The  ITT  system  to  be 
developed  under  this  concept  should  provide  a  high-fidelity  environment  for  repeated,  controlled  and  rigorous 
investigations.  Research  questions  include  instructional  strategies,  dynamic  assessment,  feedback,  and 
validating  training  effectiveness  [Tannenbaum,  '98].  A  research  plan  could  be: 

•  Test  New  Learning  Strategies  in  Scenario-Based  Training.  One  should  test  recently  introduced 
training  strategies  against  the  particular  training  tasks  of  a  real  partner  who  is  engaged  in  team  training: 
stress  exposure  training,  cross-training  to  acquaint  team  members  with  each  others’  jobs,  team  self- 
correction,  and  communication  training. 

•  Improve  the  reliability  of  measurement  and  of  feedback  by  standard  tools. 

•  Broaden  and  accelerate  adoption  of  immersive  SBT  by  enabling  it  to  be  performed  with  less- 
experienced  instructors,  via  help  for  the  hard  parts — evaluation  and  feedback. 


5.  HOW  TO  DEVELOP  SUCH  AN  IMMERSIVE  TEAM  TRAINER? 

5.1  Development  with  Training  Partners 

Our  UNC  experience  as  tool-builders  is  that  any  useful  new  system  must  be  developed  with  real  users  on  real 
problems.  For  37  years  the  UNC  computer  science  team  has  done  this  in  no-money-changes-hands 
collaborations  with  protein  chemists,  biologists,  physicists,  surgeons,  radiologists,  oncologists  [Brooks,  '96]. 
It  also  helps  to  have  two  users  of  a  new  system,  with  different  applications,  so  that  the  system  is  not  too 
specialized.  But  it  has  to  be  really  useful  to  each  of  the  users,  or  they  won’t  keep  collaborating. 

5.2.  U-Try-It  Facility 

From  the  beginning  the  project  should  maintain  two  immersive  environment  set-ups.  One  should  be  the 
Research  Laboratory,  where  new  technologies  should  be  continually  integrated  into  the  system.  The  other, 
the  Facility  for  Immersive  Team  Training  (FITT)  should  be  a  state-of-the-art  but  stable  hardware-software 
system,  a  “U-Try-It”  facility,  with  its  own  staff  experts  in  scenario-based  training. 

A  major  impediment  to  the  acceptance  and  adoption  of  immersive  system  technology  into  everyday  training  is 
the  capital  cost  (high,  today,  but  coming  down  fast)  and  the  need  for  expert  staff  to  even  try  out  the 
technology.  The  project’s  “U-Try-It”  FITT  should  be  an  international  asset  where  team  trainers  can  go  to 
develop  and  test  training  scenarios,  to  do  training  feasibility  demonstrations,  and  to  pilot  for-real  training,  with 
the  staff  experts’  help.  Providing  a  means  for  serious  inquirers  to  try  the  methodology  without  these  up-front 
investments  should  radically  accelerate  adoption. 
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The  Real  Scenario 


Some  Real  Objects,  Most  Virtual 


As  Experienced  by  Trainee  Team 


Why? 

■  Train  by  simulation? 

■  Learn  by  doing,  vs.  schoolhouse 

■  Build  team  by  shared  experiences,  stories 

■  Practice  safely,  varied,  cheaply,  often,  under  stress 

■  Capture  all  for  after-action  review 

■  Immersion  vs  monitor? 

■  Realism  produces  stress;  teaches  situation  awareness 

■  Team  skills,  body  action  (“Do”  vs.  “Say”) 

■  Teams? 


■  Team  skills  are  under-trained 


Why  Train  Under  Stress? 


■  Performance  degrades  under  stress 

■  Scerbo  et.al 15  residents,  thoracostomy 


Trained  to  100%  performance 
Under  simulated  combat: 

Just  as  fast 
Only  23%  good, 

1 7%  poor 
9  “killed”  by  sniper 
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Why  Do  High  End?  -  Training  Pyramid 


■  To  make  a  cost-effective  system, 

make  an  effective  system,  then  cost-reduce. 

■  We  need  the  whole  performance-cost  spectrum. 

a  Many  components  will  prove  effective  on 
consumer-level  platforms. 

■  Already  seeing  such  spin-offs  from  military  VEs: 

■  Microsoft’s  T-30  Flight  Simulator 

■  America’s  Army  game 

■  First  to  Fight — X  games,  $3M  private  funding 

■  Anti-Terroism  Tactical  D  S  —  Atomic  Games,  $1 M  private 


Today’s  VEs  for  Team  Training 

■  Superbly  successful  for  pilot  teams 

■  Everything  reachable  is  real 

■  Everything  virtual  is  out-the-window 

■  User  is  unencumbered,  usually  seated 

■  Pilot  and  copilot  can  share  one  viewpoint 

■  Pretty  good  also  for  some  solo  training  on  foot 

■  Quite  poor  for  dismounted  teams 
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What  It  Will  Require-Short  Form 


■  Learning  science  advances 

■  Scenario-based  stressful  training 

■  How  to  produce  scenarios  economically 

■  Simulation  technology  developments 

■  Multi-trainee  Caves 

■  Whole-body  tracking 

■  Interaction  techniques 

■  Environment  capture,  reconstruction 


FPB  et  al. 


10 


Training  Science 

■  Teamwork  skills  for  high-performance  teams 

■  Distinct  from  individual  task  skills 

“A  team  of  experts  is  not  the  same  as  an  expert  team.” 

■  Consistent  across  tasks,  domains 

■  Communication,  shared  mental  models  crucial 

■  Exploit  what  is  known,  to  make  effective  system 

■  Learn  much  more,  using  system  as  lab 


■ 

Scenario-Based  Training — Key  Strategy 

■  Repeated  practice,  lots 

■  Measurement 

■  Coaching  &  Feedback 

■  Team  self-correction 

■  Try  again! 

■  Variety  of  scenarios,  to  form  mental  templates 

■  But  scenarios,  environment  models  costly  to  produce 

■  Stress 
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Rapid,  Cheap  Scenario  Generation 

■  FAA’s  RRLOE  System  enhanced  with 
branching 

■  Model  acquisition  by  laser  and  video 

■  Scenario  capture  by  computer  vision-(way  out) 
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What’s  Technologically  Harder? 


■ 

Flight  Sims 

Dismounted 

Display 

Unified 

Distinct  for  each  user 

Tracking 

None 

Head  and  limbs  for  each 

Locomotion 

None 

Extensive,  each  distinct 

Tools,  instruments 

Physical  mockup 

Physical  and  virtual 

Interaction-users 

Speech 

Speech,  touch,  face 

Interaction-world 

Vehicle  controls 

Various  manipulanda 

Objects 

Real  touch 

Physical  and  virtual 

People 

Speech 

Speech,  body  language 

Physics 

Vehicle  physics 

Many  moving  objects 

Sound 

Unified  for  all 

Localized  for  each 

Needed  Immersive  Technology 

■  Whole  team  in  one  space 

■  See,  touch,  talk  to  each  other 

si  Personalized  stereo  views  of  VE 

■  No  encumbrances  beyond  “glasses” 

■  Real  instruments,  tools,  intermixed  with  virtual 

■  Useful  for  team  training — Everybody  tracked 
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An  Immersive  Display  Approach 


■  CAVE™ -I  ike 

■  Ceiling-mounted  projectors  projecting  outward 

■  4  eyes,  then  8  eyes,  with  individual  images 
□  PC  cluster  with  graphics  processing  cards 
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Tracking 


■  Wide-area  full-body  tracking 

■  Tracking  of  tools  as  well  as  people 

■  Many  approaches — result  will  be  hybrid 

■  New  theoretical  approach  for  design 

■  Acoustic  spread-spectrum  tracker 

■  Video  tracking  with  sub-imposed  skeletal  models 

■  Exploit  tracking  data  for  A-A  review 

■  “Pinball  scores”  for  self-training 
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Interaction 


■  Mixed  real  and  virtual 

■  Objects 

■  People  (autonomous  agents) 

■  Collision  detection 

■  Proper  physical  responses 

■  Novel  interaction  techniques 

■  Imperceptible  redirection-walking,  hand 

■  Passive  haptics  (e.g.,  ledge  floor  in  pit) 
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Getting  Such  Systems  Adopted 

■  Develop  with  two  real  users 

■  Demonstrate  effectiveness  with  real  users 

■  Make  trying  it  easy  and  cheap  for  lots  of  users 
■  U-Drive  It  facility  with  “shepherds” 
■Trailer-packaged  lab  for  mobility 

■  Ensure  successful  early  adopters 
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